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Introduction 


In the black-and-white world of medical terminology, qual- 
ity can be a slippery concept. Most health care 
providers recognize that quality is a good idea and 
aspire to providing the highest quality care possible to 
their patients. But how do we know that quality improve- 
ment actually correlates with better health care? To 
know for sure, we need to apply a scientific approach to 
performance measurement. 


By measuring clinical performance, you're taking an 
important step in providing better care for your 
patients.The process of quality improvement is similar to 
the process of how patient care is delivered, wherein a 
diagnosis is made, treatment administered, and follow- 
up conducted. In this case, your facility's care delivery 
system is the “patient.” Performance measurement will 
provide you with the diagnostic information you need to 
make informed improvement decisions. It is an essential 
element in any quality improvement strategy. 
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The effort is worth it. Quality improvement activities that 
follow from performance measurement lead to tangible 
benefits, including: 
e Improved patient care. 
e Greater employee satisfaction. 
e More efficient processes. 
e Opportunities to reduce costs. 
e Ability to meet regulatory requirements for state 
and federal grants. 


How? Performance measurement is a method to identify 
and quantify the critical aspects of care within your facili- 
ty. When you measure important aspects of care, you 
not only create a valuable source of data regarding your 
facility's greatest areas of competence, but also identify 
those areas that require improvement and that will pro- 
duce the greatest benefit for patients and staff when 
adequately addressed. 


In an era of shorter patient visits and increased 
demands on productivity, a system for routine perform- 
ance measurement is key. Without it, quality too often 
becomes a hit-or-miss endeavor in which staff invest 
time and energy to improve an element of care that may 
or may not be critical, and whose improvement may or 
may not have an impact commensurate to the required 
inputs. 
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Figure 1: 
Performance Measurement Step-by-Step 


On your mark. 


1. Select a quality of care indicator. 


2. Define the measurement > 
population. 


3. Define the measure. 


7. Run a pilot test. 


4. Create a data collection plan. 


Content overview 


This guide provides a step-by-step process for measur- 
ing clinical performance. The steps are organized into 
three phases as shown in Figure 1. Before you review 
them, take a moment to visualize yourself as an Olympic 
runner at the starting line of the finals for a well-attended 
track event: On your mark, get set, go! 


On your mark. 


During this phase of a race, runners ensure that they 
are in the proper lane and position themselves in the 
starting blocks. For the purpose of performance meas- 
urement, you will use this phase to select a quality of 
Care indicator, to define what to measure, and then 
decide how to measure it. 


Get set. 


At this point in a race, runners settle back into the 
blocks and mentally perform a "dry run” of the race as 
they wait for the starting gun. You will use this phase to 
document the measurement process, train your data 
abstractors, and ensure that the measurement defined 
in the first phase can be carried out as intended. 
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Get set. 


5. Develop data collection tools 
and instructions. 


6. Train the medical record 
abstractors. 


Go! 


8. Collect data. 


9. Analyze data and plan QI 


—> activities. 


10. Display and distribute data. 


11. Evaluate the measurement 
process and determine how to 
improve it. 


Go! 


Finally, runners are given the cue to explode from the 
starting blocks and put their training to work. This is the 
phase when data are collected, analyzed, and distrib- 
uted to management and staff. It is also the point at 
which planning will begin to determine the quality 
improvement activities that will follow from your perform- 
ance measurement, including whether changes should 
be adopted to improve your performance. 


Of course, just as an athlete’s career does not conclude 
with a single race, performance measurement does not 
conclude with a single measurement. An important part 
of retraining for the next race includes an evaluation of 
your process to determine whether it has worked and 
what improvements should be made the next time. The 
final section of the guide describes how to act on infor- 
mation that has been gathered and how to continue the 
measurement process. 


Taken together, the eleven steps are a manageable 
approach to performance measurement, particularly 
given the freedom you have in selecting what to 
measure. This is not a marathon race in which you must 
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measure the entire system of patient care. Rather, you 
will pick and choose those clinical processes that will 
help measure the degree to which your facility 
successfully achieves implementation of a particular 
process of care. 


How to Use this Guide 


This Guide to Clinical Performance Measurement may be 
used both as an introduction to performance measure- 
ment and as a reference tool. For example, if you are 
new to performance measurement, you may want to 
begin by reading the guide in its entirety and then go 
back to utilize the step-by-step guidelines. If you are 
already familiar with the process or have a measurement 
program currently in development, feel free to go direct- 
ly to those steps in which you need guidance. 


Limitations 


This guide has certain limitations. While it will provide 
you with a thorough and detailed approach to measuring 
Clinical performance, it does not explore every option 
for designing a measure or analyzing its results. The bib- 
liography provides a list of additional resources should 
your needs extend beyond the scope of this document. 


In addition, the guide is not a stand-alone instruction book 
for quality improvement. A patient with a fever of 101 
degrees does not begin to feel better simply because a 
nurse takes his temperature. The thermometer only serves 
to measure the condition, thereby shaping the treatment 
decisions intended to help the patient improve. 


The same applies to your facility's measurement efforts. 
Once you identify an area that requires improvement, the 
real work begins. Therefore, consider performance meas- 
urement as a tool to assist with your facility's overall 
quality improvement strategy. 


Finally, performance measurement results will not be the 
whole picture of the quality of care at your facility. Care 
delivery is shaped by a wide range of patient and provider 
variables for which no measure can be a perfect indicator. 
However, by measuring multiple aspects of patient care, 
you will have the broad brush strokes required to create a 
more complete picture of your service quality. 


The next section discusses the goal of performance meas- 
urement in more detail and provides some basic assump- 
tions upon which the process is based. Then, it’s time to 
Step onto the track for the step-by-step approach to per- 
formance measurement. 
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Goals & Assumptions 


Can you imagine beginning a race in which the finish line e Make the changes necessary to ensure that a 

constantly moved—or worse yet, in which there wasn't a larger proportion of patients receives the 

finish line at all? It is difficult to be strategic about your appropriate clinical interventions. 

measurement efforts when the end is unclear. As you 

embark on measuring your facility's performance, keep Figure 2 provides comparative data showing how vari- 

the following goal in mind. ous HIV care providers across New York State worked 
toward this goal over a two-year period (1999 and 

Goal 2000). The facilities measured four care elements: 
HAART Usage in ARV Therapy (viral load >30,000 

The goal of performance measurement is to measure copies/mL or CD4 <350 cells/mm3), PCP Prophylaxis 

and analyze data from the system in which care is (CD4 <200 cells/mm3), PPD Performance (annual), and 

delivered in a broader effort to: Pelvic Exam (annual). The percentages represent the 

e Monitor the quality of care provided. proportion of patients who received the related care as 

e Define possible causes of system problems. defined by the facility's respective measurement 
criteria. 

Figure 2: 


Comparative Performance Measurement Data 


HAART Usage PCP Prophylaxis PPD Performance Pelvic Exam 
(VL>30,000 or CD4<350) (CD4<200) (annual) (annual) 
Facility Type 1999 2000 1999 2000 1999 2000 1999 2000 
Community Health Center 67% 73% 89% 93% 78% 80% 73% 83% 
Drug Treatment Program 69% 74% 91% 95% 90% 92% 82% 91% 
Hospital Based Programs 70% 78% 94% 95% 72% 79% 80% 86% 
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1 2 Quality and Performance Measurement Goals and Assumptions 


For every measure, nearly all types of facilities showed 
improvement in levels of performance between the first 
and second year. While the measurement itself did not 
yield the improvements, the process did provide a start- 
ing point for making informed decisions about how to 
improve and track progress. Even with the positive 
results shown in Figure 2, the facilities continue to 
remeasure performance on an annual basis. Given the 
ever-changing nature of HIV health care delivery, ongoing 
assessment is key to maintaining improvement gains. 


Quality and Performance Measurement 


Some basic assumptions about quality form the founda- 
tion for any successful clinical performance measure- 
ment effort. 


First, quality improvement requires a working environ- 

ment in which the HIV program: 

e Wants to improve system performance rather than 
punish employees, 

e Engages staff in the entire process, from measure- 
ment selection through process evaluation, by 
including and informing them about every step in 
order to build support and ownership in the 
process, and 

e Provides adequate resources and leadership to 
sustain the process. 


Second, quality improvement methods and tools, includ- 
ing performance measurement, focus on system 
performance rather than individual employee perform- 
ance. Employees’ performance is largely dictated by the 
systems within which they work—everything from train- 
ing to paperwork to care delivery methods. Therefore, 
improving a particular element of care may require 
altering various aspects of the system. Ultimately, as 
system performance improves, so too will individual 
employee performance. 
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Performance Measurement: 


Step-by-Step 


Earlier in this guide, the eleven steps of performance 
measurement were introduced as a three-part process 
modeled upon the commands at the start of a track 
race, as shown in Figure 3. 


This section discusses the steps in detail. Each discus- 
sion begins with background information and is followed 
by a “to do list” for completing the step. Following each 
step, a shaded sidebar provides an example of how one 
HIV care provider team completed the related tasks. 
These examples do not prescribe how your facility 
should complete the step but rather demonstrate one 
way the step could be completed. 


Figure 3: 
Performance Measurement Step-by-Step 


On your mark. 


Get set. 


Aside from envisioning how to complete each step at 
your facility, you will need to consider how to assign 
specific tasks. Larger sites may choose to utilize individ- 
uals from an existing quality committee, while smaller 
sites may seek representatives from areas such as 
patient care, record keeping, and management. 


There are several organizations to help you with deci- 
sions such as these and with strategies to organize your 
quality program. Some provide quality consultation free 
of charge; others offer services and materials for a fee. 
While these areas extend beyond the scope of this 
guide, the bibliography provides quality consultant web 
sites where you can find additional information. 


Go! 


8. Collect data. 


1. Select a quality of care indicator. 5. Develop data collection tools 


and instructions. 


2. Define the measurement 5 : 
population. 6. Train the medical record 
abstractors. 


3. Define the measure. 


7. Run a pilot test. 


4. Create a data collection plan. 
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9. Analyze data and plan QI 


ae S activities. 
10. Display and distribute data. 
11. Evaluate the measurement 


process and determine how to 
improve it. 
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On Your 
Mark. 
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Step 1: Select a quality of care indicator. 


Discussion 


A quality of care indicator is an aspect of patient care 
that is measured to evaluate the extent to which a facili- 
ty provides or achieves a particular element of care. 
Generally, indicators are based on specific standards of 
care derived from guidelines issued by a professional 
society and/or government agency. 


This list is not exhaustive. There are many elements of 
HIV care and any single element may have multiple indi- 
Cators. But remember, performance measurement does 
not require that you measure your entire system of care. 
You need only measure those aspects that will help you 


determine how a particular care element is delivered to 


your patients. 
Figure 4 lists various elements of HIV care and corre- 


sponding aspects of care, which may be used as When you use any of those indicators for the very first 


indicators. time, we recommend that you start with one pilot test to 


familiarize yourself with the development and the review 


processes. 
Figure 4: 

Sample Quality of Care Indicators 

Element of HIV Care Quality of Care Indicators 

HIV staging CD4 cell count test performed every 4 months 


Viral load measurement test performed every 4 months 


Antiretroviral therapy ARV drug usage for experienced and naive patients 
HAART usage 

Tuberculosis screening Annual PPD testing 

Ol prophylaxis PCP prophylaxis (CD4<200/mm3) 


MAC prophylaxis (CD4<50mm3) 
Pneumococcal vaccination 


Gynecological care Annual pelvic exam 
STD management Annual syphilis serology 
Substance use education Annual discussion of substance use 


Annual discussion of tobacco use 


Coordination of care Annual dental exam 


EEE SS 
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| 6 Performance Measurement: Step 1: 


Select a quality of care indicator 


To do list 


Review care guidelines. 

Reference both internally and externally developed care 

guidelines during the indicator selection process. 

Federally-approved treatment guidelines for HIV and AIDS 

are available from several sources, including: 

e AETC National Resource Center (www.aids-ed.org) 

e New York State DOH AIDS Institute 
(www.hivguidelines.org) 

e HIV/AIDS Treatment Information Service 
(www.hivatis.org) 


Additional guidelines may be available at the state level. 
Externally-developed guidelines may not always corre- 
spond with your facility's desired care level; discuss 
them with your colleagues to determine if and how to 
adapt the guidelines to better fit your locally context. 


When guidelines do not exist for a particular care ele- 
ment, use your clinical team to establish a local stan- 
dard of care. 


Identify possible indicators. 

Identify aspects of care for performance measurement, 

keeping in mind three main criteria: 

e Relevance—Does the indicator relate to a condition 
that occurs frequently or have a great impact on the 
patients at your facility? 

e Measurability—Can the indicator realistically and 
efficiently be measured given the facility's finite 
resources? 

e Improvability—Can the performance rate associated 
with the indicator realistically be improved given 
the limitations of your clinical services and patient 
population? 


If you answer “no” to any of these questions, the indica- 
tor—while still relevant to patient care—is probably 
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either too difficult to measure or less than critical to 
patient care. On the other hand, if you answer “yes” to 
all of the questions, you have most likely found a viable 
indicator that will give you the most benefit for your lim- 
ited measurement resources. 


Other criteria for selecting them is the strength of eVi- 
dence that supports the efficacy of the intervention that 
they measure, whether they reflect standards of care or 
whether they measure aspects of care that are linked to 
desirable patient outcomes. 


Select the indicator(s). 

Together with your team members, make a final selec- 
tion from the list of viable indicators. Strive for consen- 
sus and, at the very least, ensure that all members are 
able to live with the decision. This will help secure their 
support throughout the measurement process. Group 
decision-making methods help teams establish priorities 
and gain buy-in among members. (Note: See the bibliog- 
raphy for source material on quality tools related to 
group decision-making.) 


Modify the indicator(s), if necessary. 

If you choose to use an externally-developed indicator 
and it does not meet the standard of practice used in 
your Clinic, feel free to modify its content as long as the 
modifications do not compromise the indicator’s overall 
relevance and measurability. An example of where an 
externally developed indicator may not reflect local prac- 
tice is when your standard is to perform a test more fre- 
quently than recommended by the statewide measure, 
such as when PAP smears are measured semiannually 
instead of annually. Another might be when your practice 
is to monitor treatment every three months but the 
statewide measure is to monitor every four months. 
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Performance Measurement: . 
Step-by-Step sow be 


Define the measurement population l / 


Step 1: Selecting a Quality of Care Indicator 


A community health care center offers a wide array of services, includi 
patients. The director of the HIV/AIDS clinic understands the importanc 
included it in the clinic’s annual goals in order to improve patient satisf 


ng primary care for over 200 HIV-positive 
e of performance measurement and has 
action and increase eligibility for state grants. 


A committee representing various work units is assem 
Committee members include a health care worker, da 
HIV/AIDS unit. 


bled to organize the effort and implement the measures. 
ta control employee, doctor, and the head nurse of the 


At the committee's first meeting, the director reviews the purpose of performance measurement and the concept of 
quality of care indicators. Then the team generates a list of potential indicators based on internally- and externally- 


developed standards of care, as well as those suggested by quality-related funding requirements and members’ indi- 
vidual performance measurement “wish lists.” 


For the final selection, the team members prioritize each indicator based on three measurement criteria: relevance, 
measurability, and improvability. Discussion initially focuses on substance use and pelvic exams. Several members 
have recently attended a conference on substance use and are eager to begin a quality improvement project on the 
topic. Other members, however, question the indicator’s relevance given the facility's relatively low drug use rate. 
Attention shifts to pelvic exams. After reconsidering the three criteria, the team selects pelvic exams as its top 
priority. 


Step 2: Define the measurement population 


Discussion 
The measurement population consists of those patients e Age: Are there particular age limits? 
who are eligible for measurement based on pre-estab- e Patient condition: Is a confirmed diagnosis 
lished criteria. Defining a population requires identifying required, or simply symptoms or signs? Do certain 
both who is a part of your population and who is not. conditions make the patient ineligible? 
e Active treatment status: How many visits are 
To do list required for eligibility? Must the patient currently be 
in treatment? Must the treatment have occurred 
Define the target population. within a certain time frame? 
Assemble your team members and brainstorm a popula- | | 
tion profile. Within the profile, be sure to specify: When you are finished aaclessims these questions iss 
e Location: What facilities within the care system will any others deemed appropriate by your team, you will 
be included? have a list of eligibility criteria. 


e Gender: Does the indicator apply exclusively to 
men or women, or to both? 


ET 
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18 Performance Measu Define the measure 


Step-by-Step 


Step 2: Defining the Measurement Population 


The week after selecting an indicator, the HIV/AIDS unit measurement team reassembles to discuss bi pigaslle: 
ment in more detail. The group discusses who the population should include and takes the following notes: 


e Patients of Health care center's HIV/AIDS unit 
e Documented HIV diagnosis 

e 

e 


Female patients > 16 years old pu ' 
Patients who have been seen by a clinician at least twice within the 12 months of the review period 


Step 3: Define the measure 


Discussion To do list 

The measure is essentially the indicator in the form of a Write the question. 

question (e.g. Was the CD4 count measured and the Phrase the indicator in the form of a question. 

result documented in the past four months?) to which 

there are a certain range of responses based on patient Determine whether any “not applicable” condi- 

documentation (e.g. Yes, No, NA). Figure 5 provides tions exist and define them. 

sample questions and response parameters for some For example, you may find that some patients are eligi- 

indicators introduced in Figure 4. (See appendix, ble to be measured by an indicator that specifies that an 

‘Sample quality of care indicators,’ for more indicator exam should be performed within a specific timeframe, 

definitions.) but that the referral was made just a few days before 
the end of the review period. Because a sufficient 

At this step it is important to define the measure by amount of time has not elapsed in which to obtain the 

Clearly documenting the “yes” and “no” responses. exam, you may wish to consider this situation an “NA.” 

Whether a test has been performed and there is docu- This group will be your NA population and consists of 

mentation to that effect, or in the case of an exam, those patients who are eligible based on criteria such as 

whether there is documentation in the record that the location, gender, age, and treatment status, but must be 

visit occurred, it is critical to specify the parameters for excluded, for example, based on treatment time frame. 

each response option. For example, you can further In some cases, your eligibility criteria will preclude the 

define the criteria for “yes” and “no” response options identification of any “NA” conditions. 


for a measure by specifying what kind of documentation 
is acceptable, the specific timeframe that is needed to 
perform the activity, or any other clinical or demograph- 
ic parameters needed to make the measure as clear 
and precise as possible. 
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Figure 5: 
Sample Question/Response Parameters 


CD4 Cell Count 

Was a CD4 count performed within the last 4 months? 

O Yes: A CD4 count of ___ was documented OR fa ee 

QO No: A CD4 count was not documented within the last 4 months of the review period. 
O NA: A CD4 count was ordered within the past week with no result documented. 


Viral Load 

Was a viral load performed within the last 4 months? 

O Yes: A viral load of ___ was documented on Ben’) Bich) Saad 

QO No: A viral load was not documented within the last 4 months of the review period. 
O NA: A viral load was ordered within the past week with no result documented. 


Antiretroviral Therapy 

Was the patient taking antiretroviral drugs at the time of the review? 

QO Yes: The patient was taking the following drugs at the time of the review: 

QO No: The patient was not taking any ARV drug(s) at any time during the review period. 


TB Screening (Eligibility: HIV- infected patients without a history of previous TB treatment or a history of a positive PPD) 
Was PPD testing performed (placed and read within 72 hours) in the past 12 months? 

O Yes: PPD screening was performed (placed and read). 

O No: PPD screening was not performed (or it was placed but results not documented). 


PCP Prophylaxis 

Did the patient with fewer than 200 CD4 cells/mm3 and no sustained CD4 cell increase > 200/mm3 during the last 6 
months of the review period receive PCP prophylaxis? 

O Yes: The patient received PCP prophylaxis. 

O No: The patient did not receive PCP prophylaxis. 


Step 3: Defining the Measure 


After defining the measurement population for pelvic exams, the committee writes the indicator in the form of a 
question: Was a pelvic exam recorded within the past 12 months? The team begins talking through the various 
responses. After listing the most obvious outcomes, the head nurse pulls a few sample charts to see if they spark 
additional response categories. The end result is the following: 


e Yes: A pelvic exam was recorded in the past 12 months. 
Pelvic Exam © ‘Date: / /_.. 
Pap Smear Date: /__ /__ 


e No: A pelvic exam was not recorded in the past 12 months. 


e NA: A pelvic exam is scheduled and the scheduled date is after the review period’s end date. 
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Step 4: Create a data collection plan. 


Discussion 


The most complete source of information on diagnosis, 
treatment, and clinical outcomes of care is the medical 
record. Different facilities store information in different 
ways. Some maintain paper medical records, others use 
software to create electronic medical records (EMRs), 
and still others use software to convert individual 
records into aggregate data. In Step 4, your task Is to 
find the most efficient way to collect information from 
your facility's records, to draw a sample of those 
records for measurement, and to select the person(s) 
who will collect the data. 


Sampling allows you to make inferences about a large 
group (total population) based on observations of a 
smaller subset of that group (sample). Most facilities utt- 
lize random sampling, wherein medical records are 
drawn from the total population in such a way that each 
time a record is selected, every record in the population 
has an equal opportunity to appear in the sample. 


As you draw your sample, keep in mind that perform- 
ance measurement is intended for evaluation, not 
research. A research project may encompass several 
years of data in order to make conclusive additions to a 
body of knowledge. But performance measurement 
based on ten year old data is hardly relevant to the qual- 
ity improvements you seek today. Therefore, it is critical 
to place a premium on obtaining the most current, com- 
plete, and accurate information possible. 


(Note: Your right to review confidential medical records 
for health care system oversight is protected by the 
Health Insurance Portability and Accountability Act of 
1996, provided that information disclosure is limited to 
the minimum necessary to achieve the purposes of the 
use. Demonstrate respect for patient confidentiality, 
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however, throughout the measurement process. For 
example, identify your data by medical record number 
rather than patient name or use aggregate data if indi- 
vidual records are not required. See bibliography for 


more information.) 


To do list 


Identify and validate existing data. 

Determine if any data related to your defined population 
have already been collected. If you have databases that 
already exist in your facility that capture the patient-spe- 
cific information you are measuring and can be queried 
to produce information in the specified timeframes, you 
may substantially reduce your workload. Investigate all 
possible sources. Aside from central administration, 
your facility may store data in areas such as the phar- 
macy or lab. If you find an existing data source, deter- 
mine the ease with which you can parse out patient 
charts. Even the most expansive database may not be 
practical if it cannot be queried for eligible patient files; 
however, an electronic medical record system may well 
contain the necessary information. 


lf you find usable data, validate its accuracy through an 
independent review of medical records. For data col- 
lected directly from record review, pull 8 to 10 charts 
from the sample to assess the reliability of the chart 
abstraction process. For computer generated data, 
review the entire report to ensure that it makes sense 
on a practical level and review a sample of the records 
to verify the accuracy of the database. 

More likely than not, you will need to collect at least 


some data on your own. The remaining steps assume 
this is true. 
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Step-by-Step 


Step 4: 


C 


Select a data collection method. 


Clinical data abstraction, the process of gleaning data 
from a larger data set, is primarily achieved through 


record review and/or administrative review. 


With record review, the abstractor collects data manual- 
ly from individual medical records, whereas with adminis- 
trative review, the abstractor gathers information from 

data previously collected by the facility's central adminis- 


tration. In the event that both options are available, 


Figure 6 provides some pros and cons for each to help 


you select the most appropriate method. 


Identify eligible records. 


Separate out the records that are eligible for measure- 


Figure 6: 
Collection Method: Pros and Cons 


Pros 
Record Ability to tailor measure to match the informa- 
Review tion requested by the committee 


Direct applicability of the question to the med- 


ical record 


Immediate response to findings in medical 
record 


Learning experience through peer review 
process 


Inter-rater reliability procedures provide poten- 


tial for thorough validation of data* 


Administrative Universal data sets are available 


iew 
nee Data are easily aggregated 


Trends easily detected once reports are 
established 


Ability to link with outcomes captured by 
database 


Facilitated data manipulation (e.g. longitudinal 
file construction) 


Reproducibility of established reports 
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ment using the eligibility criteria developed in Step 2. If 
your facility tracks any of the data in a centralized data- 
base, determine if you can query it to narrow down the 
initial batch of eligible patient files. 


Draw a random sample. 

The minimum sample size required for an accurate 
measurement is based on the number of eligible cases. 
Oversampling of one group within your population may 
be required if one or more indicators in a set pertains 
exclusively to that group. For example, if annual pelvic 
exams are among a group of indicators, a larger num- 
ber of records from female patients should be reviewed 
to ensure that annual pelvic exam performance rates are 
calculated from a sufficient number of files to provide 
for adequate statistical strength. Another reason to 
oversample is to compensate when records are actually 


Cons 


Resource commitment for frequent chart 
reviews 


Potential for breach in confidentiality 


Potential difficulty in obtaining specific medical 
records 


Available information may not match that of the 
indicator 


Often requires retrospective interpretation of 
data 


Expertise in database management required 
Dependence on quality of data entry 

Limited opportunity to validate clinical informa- 
tion which is often extrapolated from coded 


information 


Requires review of medical records to validate 


information apie May. 
yr i WCE Z 
Z J <P a ‘4 
bass \ 
“Note: Inter-rater reliability is discussed in detail in Step 7 ; fa -‘ ~ ) # 
a, 
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Figure 7: 
Minimum Sample Table, HIVQUAL Improvement Project 


ial bee, ize: Women Total 
Lida sata as cits HS Pull ean ee Pull Minimum Size 

50 or fewer 11 19 30 
51-75 12 22 34 
76-100 13 23 36 
101-125 13 25 38 
126-150 14 25 39 
151-175 14 26 40 
176-200 . 14 27 41 
201-225 14 27 4l 
226-250 14 27 41 
251-275 14 28 42 
276-299 14 29 44 
300 or more 5 30 45 


ineligible or are unavailable for review. 


Figure 7 lists the minimum sample sizes calculated for Select the data abstractor(s). 

the New York State Department of Health AIDS Institute Decide who will collect measurement data from the sam- 

HIV Quality of Care Program. You may use these figures ple. If possible, engage providers in a peer-review 

or calculate a sample size based on your own facility's process to broaden the sense of ownership and learn 

requirements. from each other. At a minimum, abstractors should be 
familiar with: 

Select the required number of records required from the e Medical record. 

total eligible records using a random process. Be aware e Relevant terminology. 

that each medical record should have an equal chance 

of being included in the sample. Most spreadsheet pro- Other factors to consider are employee availability and 

grams offer random number tables. See Appendix for an comfort level working with data. 


example of a random table. 


ee em 
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Step 4: Creating a Data Collection Plan 


The measurement team reconvenes to develop a data collection 
within a report printed from the administrative database. After a 
tracks whether a patient received a Pap smear and GC culture. | 
€xam was performed, only when the test results became availa 
basic question, the group decides to use record review as the 


plan. To begin, the members look for usable data 
quick review, they realize that the database only 
n addition, it does not indicate when the pelvic 
ble. Since this is insufficient to answer the team's 
primary collection method. 


Next, the team members identify the number of all women in the HIV/AIDS unit meeting the team’s eligibility criteria. 
The head nurse runs a report and lists 78 eligible records using patient ID numbers. She reviews the table (Figure 
7) and determines that 36 records will need to be reviewed to ensure an adequate sample size. 


To select which 36 records to select from the cases out of the total of 78, the team uses a random number list 
technique. One member writes down 36 numbers between 1 and 78 from a random numbers list without repeating 
any numbers. Another selects the medical records that correspond with the random number list. 


Finally, the group discusses who will collect the data and decides to involve the medical providers in this process. 
However, one provider cautions that given the high caseload of patients in the clinic, some providers might not find 
time to collect data. The group decides to speak with the clinic leadership to free up some time for those providers 
who are involved in the collection process, and requests that 30 minutes be set aside every month for chart review. 
The clinic leadership understands the importance of collecting data for quality purposes and agrees with this plan. 
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Get Set. 


———— 
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Step 5: Develop data collection 
instructions 


Discussion To do list 


By now, you have invested a significant amount of time 
and resources to specify a critical quality of care indica- 
tor and a corresponding measurement. To ensure that 

the measurement is conducted as intended, it is impor- 


Document review criteria. 

Create a paper or electronic document listing your 
review criteria. Be sure to include eligibility criteria, and 
question/response parameters that define the “yes” and 


tant to write detailed instructions to guide abstractors “no” responses, and N/A conditions. 

through the data collection process. Written instructions 

also provide an opportunity to stress an important point: Create a data entry form. 

performance measurement is not research. Abstractors Develop a paper or electronic data entry form on which 
have an obligation to flag questionable care delivery and abstractors will record information during the collection 
to ensure that identified problems are addressed imme- process. The form should be straightforward and con- 
diately. 


cise to facilitate accurate data collection as well as any 
future data compilation efforts. Software packages such 
as HIVQUAL2 provide electronic input screens, unique to 
specific indicators (see Figure 8). 


Figure 8: 


é 


: 
[Gynecolgy Exam ze ce 


HIVQUAL2 Input Screen for | ie 
ae Exam Gynecology exam performed ? fYes sf 


Pap smear performed ? [Yes 7] 
Abnormal result ? [No y] 


Gonorrhea culture performed ? [Yes 7] 


Positive result ? No z 4 . 


Chlamydia screen performed ? [Yes 7] 


Positive result ? |Yes Ma 


Positive screen treated ? hed J 
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Step 5: Developing Data Collection Instruments 
The team documents its review criteria on the following form: 


Indicator: Annual Pelvic Exam 


A) Eligible Population: 


Patient in Health care center's HIV/AIDS unit: =O Yes OF No MR: ere Yee 
Documented HIV Diagnosis: oO Yes Oo No Date o i ey eeney = 
Female patients > 16 years old: o Yes O No Reviewer: ____ 
Two visits within the past twelve months: oOo Yes Oo No 


Only proceed if all criteria are met! 


B) Was a pelvic exam recorded in the past 12 months? 
oO Yes (A pelvic exam was recorded in the past 12 months) 
Date of Exam: _.- /___ 7. 
Pap Smear: O Yes O No (inthe past 12 months) 
O No (A pelvic exam was not recorded annually) 
oO NA(A pelvic exam is scheduled and the scheduled date is after the review period's end date) 


C) Notes 


Step 6: Train the abstractors. 


Discussion 

Written review criteria have the greatest impact when tors through the process. Clearly indicate patient eligibili- 

explained in person. During a brief training session with ty criteria and where to find required information in 

your abstractors, you not only have the opportunity to patient charts. In addition, emphasize the importance of 

review the measurement process, but also to provide responding to problem charts immediately. For example, 

useful context on how data collection will contribute to if a patient with fewer than 200 CD4 cells/mm is not 

the facility's overall quality improvement. on PCP prophylaxis and the count has been less than 
200 CD4 cells/mm3 for over 6 months, the file should 

To do list 


be flagged and the chart immediately forwarded to the 
appropriate providers for further review and follow-up. 
Schedule approximately 30 minutes for an informal train- 


ing session with the abstractors selected in Step 4 as If possible, allow abstractors to do a “dry run” using a 


close to the actual time of data collection as possible. sample medical record at the end of the training ses- 
Distribute the review criteria and data collection form 
before the meeting. 


sion. This may help reveal any points of confusion in 
your documentation and surface general questions 
about the process. Also by recording the time for 


reviewing each chart, you will be able to estimate your 
time requirements. 


Begin the meeting by reviewing the basic purpose of 
performance measurement and the specific clinical 
aspects of the subject under review. Next, walk abstrac- 


ees 
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Step 6: Training the Abstractors 


The measurement team meets with the providers to review the measurement process. They begin by explaining 


nce of each field in the data collection form. Next, they walk through 


0 complete each field to ens 
data sources. ure that everyone is looking at all possible 


At the end of the meeting, one provider asks how to use the “Notes” portion of the data collection form. The 
team clarifies that this space should be used to identify issues that need further Clarification, or to flag any charts 


that require follow-up to ensure appropriate patient care. The 
: ) . Ihey add that a staff member shoul 
Cases immediately for follow-up. eS ee 


step 7: Run a pilot test. 


Discussion To do list 

The purpose of a pilot test is to assess the efficiency of Pilot test 2 or 3 records in the sample. 

the measurement process and to ensure that the Select 2 or 3 records from the total sample and ask the 
abstractors are able to get the information they need abstractors to collect the required information as 

from the sample before starting the full-scale explained during the training session. 

measurement. 


Discuss pilot test results. 

Meet with the abstractors to review the results and col- 
lect feedback about the process. Tweak the measure- 
ment process as needed. 


Step 7: Running a Pilot Test 


For the measurement committee’s pilot test, one provider collects data from three medical records taken from the 
sample. When the group meets to review the data, the provider explains that patients’ risk factor data are incomplete 
due to the time required to find them. The team empathizes with the provider's frustration, but stresses the impor- 
tance of patient information for future analysis. Several team members provide hints for finding the risk factor 


more quickly. 


CE ee 
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CT ees ee 


AETC NRC + NYSDOH Al Measuring Clinical Performance: April 2002 
A Guide for HIV Health Care Providers 


Performance Measurement: 
Step-by-Step Coles 


Step 8: Collecting data 


Discussion 


Your time spent crouched in the Starting blocks pays off 
during data collection. Armed with clear instructions and 
a successful pilot test, abstractors launch from the 
starting pad into the data collection phase of perform- 
ance measurement. 


To do list 


Inform other staff of the measurement process. 
Before abstractors begin collecting data, describe the 
performance measurement process and distribute the 
collection tools to the rest of the staff. Sharing informa- 


Step 8: Collecting Data 


tion helps build support for the facility's quality improve- 
ment efforts. 


Remain available for guidance. 
During data collection, an experienced staff member 


should be available to answer questions and guide 
the process. 


Document existing data, if necessary. 

If you are incorporating existing data into your measure- 
ment, transfer the relevant information onto a separate 
data collection form at this time. 


A few days before the first measurement session, the director sends an e-mail to the entire staff with a brief history of 
the performance measurement project and an outline of what employees should expect during the measurement 
cycle. The director also announces the project in a staff meeting to emphasize the importance of this aspect of care, 
particularly the benefits to patients. Finally, the director passes out a copy of the data collection form and assures 
staff members that they will be the first to receive measurement results in a meeting following data collection. 


Subsequently, each provider is assigned 9-10 charts of the 36 files and provided with an experienced staff contact 
should questions arise. They collect the pelvic exam data in the following week. 


Step 9: Analyze data 


Discussion 


When data collection is complete, data analysis begins. 
The review of your performance data is a starting point to 
identify whether care meets or exceeds the desired quali- 
ty level and to define possible causes of performance 
problems, if they exist. These activities are the initial 
steps of the quality improvement planning process. They 
enable you to prioritize your work and select the areas 
for improvement that will be addressed by the team. 


To do list 


Review data. 

As you review the raw data, keep the results in perspec- 
tive. Deficient performance measurement data for one 
indicator do not necessarily reflect the quality of care 
provided by the facility as a whole and may be caused 
by secondary factors, such as access to equipment or 
other resources, specific to that aspect of care. 


eee 
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Step-by-Step Analyze data 


Step 9: Analyzing Data and Planning QI Activities 


The measurement team meets over lunch to calculate and discuss preliminary results: 


Pelvic Exam? e Yes: 65% 
e No: 34% 
e NA: 1% 


Pap Smear? e Yes: 88% 
eNo: 12% 


The team agrees that it would like to improve the 65% success rate for pelvic exams. They decide to convene a per- 
formance improvement team to continue the process of quality improvement by identifying the possible causes for this 
rate of performance. They begin a series of follow-up meetings. In the first meeting, they develop a project plan for 
reducing their patients’ refusal rate for pelvic exams. The plan includes developing a patient and provider education 
program and establishing key milestones for the next activities. 


Compile and analyze longitudinal data, 


if possible. 

Longitudinal data studies show changes in performance whether there are disparities between them in 

over a period of time. If historical data are available, lon- receiving care. 

gitudinal studies are useful in tracking upward and down- 

ward trends in clinical performance. Long-range trend Quality tools such as run charts and control charts are 
analysis can be useful in strategic planning and manage- helpful in displaying longitudinal data and detecting per- 
ment as well as research and evaluation, while short- formance trends. This graphic display can effectively 
range trend analysis can be used to trigger quality present information and portray changes over time. See 
improvement activities. You may also want to analyze Step 10 and Figure 9. (Note: For additional information, 
your data by different patient groups to determine see the bibliography for source material on quality tools.) 


step 10: Display and distribute data 


Discussion 


To help galvanize support for the performance measure- 
ment process, it is important to summarize and display 
your data and tell people what it means. Step 10 


describes how to get the word out about your perform- 
ance measurement results. 
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Figure 9: Measurement Data in Graphic Form. 


Run Chart: Pie Chart: 
Annual PPD Rate 
on —=— 5 Did the patient receive PCP prophylaxis? 
90% —— _ 
2% 
NA 


a Sie. ae ek oe SS Se ai ae SS 
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 


_ Control Chart: 


Waiting Time 


9/16/01 
9/17/01 
9/18/01 
9/19/01 
9/20/01 
9/21/01 
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Display and distribute data 


Histogram: 


Reasons for No GYN Exam in Medical Record 


60% 
50% 
40% 
30% 
20% 
. | ' . 
0% GYN exam  GYN exam Referral | GYN exam Medical Other 
not done done, but made, but offered record not 
results not GYN appt. but patient found 
documented _ not kept by refused 
patient 


Identify the target audience. 

Your measurement team members and staff should 

be among the first to receive the results. However, 
several groups may be interested in your performance 
measurement data including health care providers, 
consumers, governing boards, and governmental 


agencies. Tailor your message to meet individual audi- 
ence needs. 


Summarize/display data. 

Whenever possible, put your results into graphic 
forms such as tables, bar charts, or pie charts. 
Graphic data displays help to convey outcomes at-a- 
glance. Use text sparingly for background and/or 
explanatory information. Figure 9 provides five exam- 
ples of how data may be displayed graphically. 


Present data. 


By presenting results in person, you will have the 
opportunity to discuss and clarify problem areas as 
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Pareto Chart: 


Which Adherence Interventions Work? 


100% 


200 100% 
160 80% 
120 60% 
80 40% 
50 
20% 
40 i 42 
24 

0 # wy lo om 4 Q% 

Face- Face- Attendance Attendance Handing Other 

to-face to-face of adher- of peer out of con- 

conversa- conversa- ence work- adherence sumer 

tion with tion with shop by group educational 


clinician nurse patient brochure 


well as to explain the next steps in the process. 


When planning the presentation, write down the one or 
two most important items you'd like to communicate 
during the meeting. Then create a presentation that 
helps you reach your goal, keeping in mind peoples’ 
busy schedules and relatively short attention spans. 
Utilize your measurement team to test different presen- 
tation strategies and find the most effective style for 
your audience. 


Use every opportunity to share and discuss your meas- 
urement data and improvement plans. For example, if 
you have a 3-month process improvement cycle, hold a 
status meeting after one month to keep people 
informed. 
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Step 10: Displaying and Distributing Data 


After measurement results are com 


piled, the team meets to discuss with whom to share th 
HIV/AIDS unit staff, the group decid re them. Aside from the 


es to provide hospital administration with the measurement data. 


The centerpiece of the team’s data display is a percentage distribution table: 


Indicator: Annual Pelvic Exam 


er 


Response Percent 
Yes: A pelvic exam was recorded within the past 12 months. 65% 
No: A pelvic exam was not recorded within the past 12 months. 34% 
NA: A pelvic exam is scheduled and the scheduled date is after the review period's end date. 1% 


The team decides to present the results at two different meetings. They meet with clinical employees first to con- 
gratulate them on the current performance level and begin discussing improvement strategies. Employees brain- 
storm ways to reduce the cases in which a pelvic exam was not recorded. Several employees point to patient 
refusal as one possible target. 


Next, the measurement team develops a brief presentation of its findings and its improvement plans for the admin- 
istration’s annual budget meeting. A reviewer offers feedback at a team practice session, suggesting that members 


“sell” the tangible benefits of performance measurement. During the meeting, members discuss how long-term per- 
formance measurement can lead to reduced costs and improved patient care in the HIV/AIDS unit. 


step 11: Evaluate the measurement 
process and determine how to improve it 


Discussion 
Your performance measurement data are only as good between two or more abstractors. In this case, the 
as the process from which they are collected. Make instrument includes both the review criteria and data col- 
sure the process works by assessing its reliability and lection form. (See the bibliography for source material 
effectiveness following each performance measurement on reliability.) 
cycle. 
To do list 
Reliability is defined as the consistency of an instru- 
ment’s measurements when used under the same condi- Solicit process feedback. 
tions with the same subjects. The reliability of a meas- Ask abstractors to evaluate what aspects of the process 
ure is important for ensuring comparability of results worked well and what posed challenges. 


over time. Frequently used in medical record abstrac- 
tion, inter-rater reliability is defined as the reliability 
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While additional information on reliability testing Is 
beyond the scope of this document, the bibliography 
lists source material on statistical analysis. 


Test inter-rater reliability. 

To test inter-rater reliability, examine both the degree to 
which values recorded by different reviewers are the 
same and the extent to which there is agreement on the 


final scores Improve the measurement process. 


Based on abstractor feedback and reliability data, make 
any necessary changes to the process before the next 
measurement effort. This may require that you modify 
the process and its related forms, retrain abstractors, 
and/or reconsider the chosen indicator as a critical 


For example, consider a project in which an abstractor 
reviews 50 patient files for a given indicator. The 
abstractor records a “yes” or “no” response for each 
file. At the end of the measurement process, the evalua- 
tor repeats the data entry process using the same set aspect of care. 
of charts. The two abstractors have matching “yes” 

responses for 45 of the 50 charts, a 90% agreement, 

and 100% agreement for the “no” response catagory. 


Step 11: Assessing the Measurement Process and Determining How to Improve It 


After the team’s presentation to the hospital administration, team members facilitate a series of follow-up meetings. 
in the first meeting, HIV/AIDS unit staff members develop a project plan for reducing their patients’ refusal rate for 
pelvic exams. The plan includes key milestones and status meeting dates. 


Team members meet to evaluate the measurement process. In preparation, one member conducts an inter-rater reli- 
ability test. Results show that while agreement between individual values and final yes/no tallies are high, there is 
room for improvement in the “yes” category. The team brainstorms potential causes and decides to develop a more 
detailed data collection form to improve the abstraction process. 


Finally, the members celebrate their considerable successes: the establishment of a performance measurement 
infrastructure and the potential for improved gynecological care in the HIV/AIDS unit. 


Youre on your way! 


Consider what you've accomplished by completing the e Evaluated the performance measurement 
11 steps of performance measurement. You process. 
e Defined a quality of care indicator and determined e Established an infrastructure for future measurement. 


how to measure it. 


e Documented a data collection plan. But don't stop now. Performance measurement slows to 


e Analyzed and summarized performance data. walk if you don’t act on the information you've collected 
e Shared performance data with internal and external and remeasure targeted indicators. The final section of 
groups. 


this guide explains some important next steps. 


ac 
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Next Steps 


Performance measurement is an integral part of quality The key to quality improvement is identifying causes 

improvement, but it is only that—one part. Now that you affecting your performance and changing systems of 

have some hard data upon which to make informed care to make improvements. For example, in the case of 

improvement decisions, make the incremental changes the HIV/AIDS unit discussed in the previous section, 

needed to improve clinical performance, remeasure, and patients’ relatively high refusal rate has a significant 

track your progress. impact on the unit's overall success rate in performing 
pelvic exams and PAP smears. Their decision to devel- 

Remeasurement Op a patient education program about the importance of 
pelvic exams is a first step in improving their perform- 

It is important to remeasure your indicator once you ance, and becomes the immediate focus of their quality 

have implemented system changes. This helps to identi- improvement activities. 

fy any incremental performance improvement or decline 

between measurements, as well as to track perform- If you’re lucky, you already have a hunch as to what your 

ance trends. Usually, you will want to re-measure your specific causes are and can begin consulting with your 

performance at least annually, although you may decide staff to address them. 

that a shorter interval is warranted, depending on your 

performance. The implementation of a quality improvement project 
requires a new set of tools. The bibliography lists 

Quality Improvement resources that will get you started. But remember, at 
this point you have mastered the measurement of Clini- 

Once you have successfully accomplished your first goal cal performance and this knowledge will help you to 

to measure one specific aspect of care, the next task is track your improvements over time. 


ahead of you. If your performance measurement data 
lead you to pursue opportunities for improvements, you 
need to start a new race. 
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i ee 
Sample quality of care indicators 


MAC Prophylaxis (Eligibility: CD4 cells < 50 and no sustained CD4 cell incr 


Did the patient receive MAC prophylaxis? 
QO Yes: The patient received MAC prophylaxis. 
QO No: The patient did not receive MAC prophylaxis. 


€ase to > 50 during the last 3 months of review period) 


Pneumococcal Vaccination 


Was the patient given a pneumococcal vaccination? 

O Yes: The pneumococcal vaccination was givenon__/__/_., 

O No: The pneumococcal vaccination was not given. 

O NA: A pneumococcal vaccination is scheduled and the scheduled date is in the future. 


Annual Pelvic Exam 


Was a pelvic exam performed within the past 12 months? 
O Yes: A pelvic exam was recorded. 
O Pap done (Abnormal? O Yes O No If abnormal, 2nd Pap or GYN referral? O Yes O No) 
QO No: A pelvic exam was not recorded within the past 12 months. 
O NA: A pelvic exam is scheduled and the scheduled date is in the near future. 


Annual Syphilis Serology 


Was a syphilis serology performed within the past 12 months? 
O Yes: A syphilis serology was performed. 
O No: A syphilis serology was not performed within the past 12 months. 


Annual Discussion of Substance Use 


Was substance use discussed with the patient during the past 12 months? 
O Yes: Substance use was discussed. Results identified: 
O Current user (within 6 months): 
O Heroin (O Injected O Intranasal O Smoked) 
O Cocaine (O Injected O Intranasal O Smoked) 
O Alcohol 
O Pills 
O Marijuana 


If a substance was injected, was safer injection/needle exchange addressed? O Yes O No 


Patient in treatment during review period? O Yes O No 


If No: If Yes, or if referral made: 

O Treatment discussed; no referral made O Detox O Outpatient, not 
O Treatment discussed; referral made methadone 

O Treatment not discussed O Methadone O Residential treatment 


O Past user only (last use over 6 months): 
Months since used: 6-12 O 13-24 O Over 24 
Prevention/ongoing treatment discussed? O Yes O No 
QO No current use (within 6 months) or past use (over 6 months) identified 
O No: Substance use was not discussed with the patient during the review period. 


Annual Discussion of Tobacco Use 


Was tobacco use discussed with the patient during the past 12 months? 
O Yes: Tobacco use was discussed. 
O No: Tobacco use was not discussed. 


Annual Dental Exam 


Was a dental exam performed within the last 12 months? 

O Yes: A dental exam was performed on __ /__ /_. 

O No: A dental exam was not performed. 

O NA: A dental exam is scheduled and the scheduled date is in the near future. 
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Random Number Calculations: Women Eligible Cases=276-300; Charts to Pulle4g 


_ *, 16, 23, 28, 33, 50, 53, 54, 60, 61, 68, 73, 77, 78, 96, 113, 117 
Eligible Cases=up to 50; Charts to Pull=29 125, 126, 129, 135, 138, 146, 152, 153, 167, 175, 177, 185, 187 


1, 2, 4,6, 7, 10, 13, 15, 17, 18, 21, 22, 24, 25, 26, 28, 30, 33, 34, 189, 193, 202, 203, 207, 210, 222, 224, 227, 228, 255, 256, 259, 
35, 37, 39, 40, 42, 44, 45, 46, 47, 49 265, 267, 270, 292, 297 


Eligible Cases=51-75; Charts to Pull=34 
1, 2, 6, 8, 11, 12, 13, 15, 16, 19, 20, 21, 24, 27, 28, 29, 30, 34, 37 


Ran ions: 
40, 41, 46, 48, 49, 50, 51, 55, 59, 61, 65, 66, 70, 73, 75 dom Number Calculations: Men 
Eligible Cases= 4 
Eligible Cases=76-100; Charts to Pull=38 xy : eI = nee php 
1.3, 7,10, 13,17, 22, 23, 24,27, 28, 33, 34,49, 48, 53,55, 57 4,8, 10, 11, 14, 17, 18, 21, 26, 27, 28, 29, 30, 32, 38, 40, 42 
* 
a 2 = 63, 64, 65, 66, 69, 72, 73, 74, 75, 82, 84, 85, 88, 94, Eligible Cases=51-75; Charts to Pull=21 
97, 5, 14, 18, 25, 26, 28, 31, 36, 42, 43, 46, 47, 49, 54, 56, 66, 69, 70 
71, 73, 75 


Eligible Cases=101-125; Charts to Pull=40 

4,9, 10, 12, 14, 15, 17, 21, 25, 26, 28, 35, 38, 39, 41, 42, 43, 44, 
47, 51, 56, 60, 61, 71, 76, 78, 86, 89, 90, 91, 95, 96, 103, 104, 
105, 107, 109, 110, 116, 123 


Eligible Cases=76-100; Charts to Pull=22 
1, 4, 13, 16, 20, 26, 33, 38, 40, 42, 46, 53, 59, 63, 64, 65, 75, 82, 
83, 87, 94, 97 


Eligible Cases=1 26-150; Charts to Pull=42 
5, 7, 8, 9, 15, 19, 24, 25, 28, 30, 32, 33, 35, 55, 57, 58, 64, 67, 80, 
81, 83, 84, 90, 91, 93, 94, 106, 110, 111, 112, 113, 116, 117, 120, 
122, 123, 126, 132, 134, 145, 149, 150 


Eligible Cases=101-125; Charts to Pull=23 
3, 4, 14, 19, 21, 24, 27, 32, 33, 36, 38, 42, 44, 47, 56, 74, 84, 93, 
104, 108, 113, 118, 121 


Eligible Cases=126-150; Charts to Pull=23 
1, 12, 16, 19, 28, 29, 32, 35, 38, 42, 43, 45, 61, 72, 85, 89, 96, 97, 
99, 106, 108, 129, 130 


Eligible Cases=151-175; Charts to Pull=44 
2, 6, 10, 11, 17, 19, 20, 21, 24, 32, 37, 39, 40, 42, 47, 61, 63, 65, 
67, 74, 79, 84, 87, 88, 96, 99, 103, 104, 108, 109, 118, 120, 127, 


130, 131, 133, 142, 147, 149, 152, 154, 165, 167, 172 Eligible Cases=151°175 Chae torneo 


2,5, 10, 12, 18, 27, 38, 46, 48, 60, 72, 74, 76, 85, 92, 94, 120, 


Eligible Cases=176-200; Charts to Pull=45 128, 129, 141, 162, 163, 165, 170 


4, 11, 25, 32, 45, 50, 52, 55, 64, 65, 67, 71, 76, 81, 82, 83, 86, 89, 
90, 92, 96, 103, 104, 105, 107, 111, 117, 119, 123, 128, 129, 131, Eligible Cases=176-200; Charts to Pull=24 
143, 159, 162, 163, 164, 165, 167, 168, 173, 177, 186, 191, 198 3, 6, 14, 37, 39, 54, 59, 75, 93, 101, 118, 122, 128, 131, 139, 143, 


148, 168, 173, 174, 179, 186, 187, 188 
Eligible Cases=201-225; Charts to Pull=46 


5, 11, 28, 29, 34, 38, 46, 47, 48, 52, 60, 80, 92, 99, 101, 104, 109, Elise Gaseeb01 595 Chane se ane 
117, 121, 126, 127, 134, 135, 136, 137, 141, 145, 149, 151, 158, 5, 12, 14, 28, 37, 43, 47, 51, 80, 82, 85, 115, 195, 195 aby Aes 
165, 168, 169, 170, 174, 180, 181, 184, 186, 193, 202, 204, 205, 173, 179, 195, 204, 219, 221, 222. 223-204 
219, 220, 222 

Eligible Cases=226-250; Charts to Pull=25 
Eligible Cases=226-250; Charts to Pull=47 4, 17, 49, 61, 67, 69, 76, 78, 81, 126, 145, 147, 157, 167, 180, 
2, 11, 14, 28, 29, 32, 40, 43, 45, 47, 60, 80, 85, 88, 89, 100, 102, 190, 194, 196, 208, 215, 224, 226, 232, 238, 248 
103, 105, 119, 121, 131, 146, 158, 159, 167, 169, 176, 180, 184, 
185, 190, 192, 196, 199, 203, 211, 213, 218, 220, 221, 230, 232, Eligible Cases=251-275; Charts to Pull=25 
ai oa 4, 17, 49, 61, 67, 69, 76, 78, 81, 126, 145, 147, 157, 167, 180 


190, 194, 196, 208, 215, 224, 226, 232, 238, 265 
Eligible Cases=251-275; Charts to Pull=48 


1, 3, 21, 29, 34, 35, 47, 63, 78, 83, 84, 85, 87, 88, 90, 91, 103, Eligible Cases=276-300; Charts to Pull=25 
116, 144, 145, 156, 157, 161, 167, 168, 180, 183, 189, 7, 22, 32, 53, 57, 68, 69, 80, 90, 94, 110, 112, 124, 135, 142, 159, 
190, 204, 208, 210, 212, 227, 232, 236, 237, 238, 239, 243, 252, 174, 188, 237, 239, 255, 278, 283, 287, 288 


257, 259, 260, 261, 268, 271, 272 
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EE ee nna 


: April 2002 
Measuring Clinical Performance: 
ea aaa A Guide for HIV Health Care Providers 


ah 


